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ABSTRACT 



This final report describes activities and achievements of a 
two-year qualitative study to determine factors that contribute to successful 
campus implementation of student outcomes assessment by evaluating 15 student 
outcomes assessment pilot projects in the California State University system. 
Data sources included the evaluation reports from each of the pilot projects, 
interim and final project reports, interviews with project directors, and 
some site visits. Factors most commonly associated with successful 
implementation of assessment included: measurement adequacy, faculty 
involvement, administrative support, and expertise of the project director. 
Following the executive summary, individual sections of the report present a 
project overview and describe the project's characteristics, results, 
evaluation, and plans for continuation and dissemination. The bulk of the 
report consists of five appendices, including a list of the project's 
reviewers, newsletter reports of the project, a research report titled 
"Evaluation of Student Outcomes Assessment Pilot Projects in the California 
State University" (by Matt L. Riggs and Joanna S. Worthley) , and the agenda 
of a 1991 assessment seminar. (DB) 
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B. Summaries 



Brief Project Description 

The CSU Institute for Teaching and Learning conducted a two-year study to 
determine factors that contribute to successful campus implementation of student 
outcomes assessment, through an evaluation of fifteen student outcomes assessment 
pilot projects in the California State University system. A preliminary outline of the 
study was sent to 25 national and state assessment leaders for comments, and a 
national steering committee consisting of experts in the field was formed to guide the 
project. A faculty research team at CSU San Bernardino conducted the study, which 
found a number of variables common to successful outcomes assessment projects on 
different campuses and in different disciplines. Factors most commonly associated 
with successful implementation of assessment included measurement adequacy, 
faculty involvement, administrative support, and expertise of the project director. The 
results of the study were widely disseminated through presentations at a statewide 
faculty seminar and several national conferences as well as publication of two 
newsletters, a booklet, a monograph, and a video tape. 

Helen R. Roberts, Project Director, Institute for Teaching and Learning, California 
State University Office of the Chancellor, 400 Golden Shore, Long Beach, C A 90802, 
(310) 985-2607 

Project Reports and Products: 

"Assessment in the CSU." Special inserts in the CSU Institute for Teaching and Learning 
Newsletter, Winter 1990 and Spring 1991. 

"Evaluation of Student Outcomes Assessment Pilot Projects in the California State 
University." FIPSE project report by Matt L. Riggs and Joanna S. Worthley, 
Department of Psychology, California State University, San Bernardino, 1992, 56 
pages. 

Student Outcomes Assessment. Sixth booklet in the "Academic Challenges" series, in 
press at the California State University Office of the Chancellor, 1993, approximately 
25 pages. 

Student Outcomes Assessment: What Makes It Work ? - Assessment Practices and 
Experiences in the California State University. Monograph published by the California 
State University, Institute for Teaching and Learning, 1992, 84 pages. 

Student Outcomes Assessment: What Makes It Work? Videotape produced by the 
California State University, Institute for Teaching and Learning, 1992, 16 minutes. 
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Executive Summary 

There have been many studies on state policy regarding student outcomes assessment, 
but few have examined the conditions for successful implementation of outcomes 
assessment at the campus level. From 1986 to 1989, the California State University 
system invested more than a half million dollars in campus pilot projects to 
demonstrate how student outcomes of various kinds can be used to assess the 
effectiveness of general education or baccalaureate degree programs. In 1989, the 
Fund for the Improvement of Postsecondary Education provided a grant to the CSU 
Institute for Teaching and Learning to conduct a comprehensive evaluation of the 
fifteen campus pilot projects, to determine factors that contributed to successful 
campus implementation of outcomes assessment, and to disseminate the results 
nationally. 

In selecting the questions to be addressed in the evaluation study, the Institute for 
Teaching and Learning consulted with assessment experts in twenty-five other states 
and national organizations. A national steering committee was appointed to guide 
the study design, interpretation of findings, and dissemination plan. The study was 
conducted by a team of faculty researchers at CSU San Bernardino. Overall project 
direction and coordination of dissemination activities was handled by the Institute for 
Teaching and Learning in the CSU Office of the Chancellor. 

The evaluation study was a qualitative examination of variables associated with 
successful implementation of student outcomes assessment on the pilot project 
campuses. The research team looked at environmental factors (such as faculty 



involvement, project director's expertise, administrative support, campus policies) as 
well as methodological factors (such as specificity of project goals, adequacy of 
measures used, cost effectiveness); and assessed these factors in relation to both direct 
and indirect project outcomes. Data sources included the evaluation reports from 
each of the pilot projects, interim and final project reports, interviews with project 
directors, and, in some cases, site visits. 

The study identified four major factors associated with successful implementation of 
assessment programs: (1) adequacy of assessment measures-including multicultural 
efficacy; (2) existence of administrative support; (3) level of faculty participation by all 
members responsible for implementation of any aspects of assessment programs; and 
(4) assessment project director's training /experience in assessment procedures. 
Relationships of numerous other variables to successful implementation of assessment 
were analyzed and reported. 

Results of the study were disseminated and discussed throughout the California State 
University and the nation by way of: 

• Two special inserts on "Assessment in the CSU" sent to more than 20,000 faculty 
members in the California State University with the ITL Newsletter. 

• A statewide one-day seminar, "Toward a New Paradigm: Assessment in 
Multicultural California," held in conjunction with the sixth AAHE Conference on 
Assessment in Higher Education, attended by nearly 100 campus representatives. 

• Presentations made by the research team at national conferences of the American 
Association for Higher Education, the Society for College and University Planning, 
and the American Educational Research Association. 

• National distribution of a monograph (1,000 copies) and companion videotape 
entitled Student Outcomes Assessment: What Makes It Work? 

• Publication of the study as the sixth booklet in CSU's "Academic Challenges" 
series, a series which reports on the evaluated results of academic pilot projects at 
CSU campuses. 

Difficulties encountered during the project included coordination of meeting 
schedules for the national advisory committee, identification of a pool of prospective 
researchers to conduct the study (only one research team responded to our RFA), and 
production timeline delays in the preparation of the various materials for publication. 

Among the most useful outcomes of the project were: 

• Reconfirmed the importance of faculty involvement, administrative support, 
measurement quality, and faculty development in the assessment process. 

• Contributed to the knowledge base on "assessment of assessment." 

• Continued improvement in CSU's relationship with the Western Association of 
Schools and Colleges and with California policy makers as they became more 
aware of CSU's commitment to examining outcomes. 
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Better dissemination of the campuses' experience than would have been possible 
to achieve without the FIPSE grant, both because of FIPSE's cachet, which helped 
attract attention to the study, and because of the resources that the project made 
available. 

Expanded and developed the network of people sharing expertise in student 
outcomes assessment, both within California and nationally. 



C. Body of Report 



Project Overview 

• A two-year study was funded by FIPSE to determine factors that contributed to 
successful implementation of 15 campus pilot projects on student outcomes 
assessment in the California State University. A $63,265 grant was awarded for 
the first year starting September 1, 1989. A continuation proposal for the 
second year of the project was submitted to FIPSE in May 1990 and a $91,814 
(including carry-over funds from year 1) grant was awarded. A request for a 
no cost time extension for the project was submitted to the program officer and 
was approved to extend the project period until March 31, 1992. 

• National and state leaders in student outcomes assessment were contacted 
before the study started for comments on study design and major focus (see 
Appendix A for the list of the people contacted). A group of experts in the field 
of assessment formed the project's national steering committee (see Appendix 
B). Guidelines and advice provided by these scholars and administrators were 
instrumental in each major phase of the project: from the design of the study, 
to the interpretation of the findings, and the strategy for national 
dissemination. 

• A faculty research team at CSU San Bernardino was identified through an RFP 
process to conduct the evaluation study. 

• Two issues of "Assessment in the CSU" newsletter inserts were published and 
disseminated together with the Winter 1990 and Spring 1991 Institute for 
Teaching and Learning Newsletter to all 22,000 CSU faculty and interested 
parties in the nation (see Appendix C). 

• The results of the evaluation study (contained in Appendix D) were widely 
disseminated through various mechanisms, including presentations at one CSU 
systemwide seminar (see Appendices E and F for the seminar agenda and 
evaluation report) and five national meetings, publication of one monograph 
(Appendix G), one videotape (Appendix H), and one booklet (in press). 
Dissemination of the project results will continue by conducting more 
presentations and discussions on assessment issues, connecting with state and 
national organizations to promote assessment practices and applications, 
publishing one more book, and selling the above mentioned project products 
through the CSU Academic Publications Program. 
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Purpose 



• The study was to evaluate 15 student outcomes assessment pilot projects in the 
California State University system to identify factors that contribute to the 
success of the projects and can be generalized across institutions and 
disciplinary areas. It was believed that the experiences learned from these 
projects are of great value in implementation and decision-making practices of 
assessment programs at higher education institutions. The results of this study 
were to be disseminated throughout the nation by various dissemination 
mechanisms. 

• Problems encountered during the project period included: (1) difficulty in 
convening the nationwide steering committee for meetings; (2) low response by 
CSU faculty researchers to the request for applications to conduct the study; (3) 
the increase of the faculty research team's work load on their campus due to 
budget reduction which in turn resulted in the delay of delivering products and 
in fewer dissemination activities carried out by the team; and (4) production 
timeline delays in the development of dissemination materials. 

Background and Origins 

• In response to the call for educational accountability and the national 
movement in student learning outcomes assessment, the California State 
University launched a series of systemwide activities beginning in 1986 to 
support educational reform through assessment. 

• In the absence of state mandated assessment, the CSU system developed in 
1988 a systemwide policy framework on outcomes assessment after two years 
of study and discussion by the chancellor-appointed systemwide advisory 
committee on student outcomes assessment. 

• Two systemwide assessment conferences were held in 1986 and 1988, 
respectively. The first conference convened CSU faculty, administrators, 
California state higher education officers, and national leaders in assessment to 
discuss various issues in assessment and how to implement assessment 
programs on campuses. The purposes of the second conference was to share 
campus experiences in assessment from their investigation and 
experimentation, and to discover the possibility of formulating systemwide 
policy recommendations. Open discussions in these two conferences were 
useful in enabling the advisory committee on student outcomes assessment to 
identify overall guiding principles and recommendations. 

• Since 1986, the California State University has supported 15 campus pilot 
projects to implement student outcomes assessment programs through the 
Chancellor's Academic Program Improvement Campus Grants Program. More 
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than a half million dollars were invested by the CSU in these pilot projects, which 
developed and field-tested a variety of assessment measures/instruments, including 
portfolios, interviews, senior /capstone projects, surveys, and examinations. The level 
of application ranged from individual course to department and the entire campus. 

Project Description 

• The evaluation study was designed and shaped by the comments from experts and 
leaders in the field across the nation. A preliminary outline for the study was sent to 
25 national and state leaders in learning outcomes assessment for comments. Nearly 
half of them responded and their comments were considered in shaping the central 
research focuses and methods of the study. 

• A group of experts in the field of assessment formed the project's national steering 
committee. Three steering committee meetings were held: (1) in January 1990 to 
establish guidelines for conducting the evaluation study; (2) in November 1991 in 
conjunction with the pilot project directors' meeting convened by the research team to 
hear the research team report the preliminary findings of the survey of pilot project 
directors, as well as to discuss the plan for the project second year and guidelines for 
the CSU faculty seminar on assessment; (3) in May 1991 to discuss the preliminary 
findings of the study and aspects related to the June CSU assessment seminar and 
other dissemination activities. 

• The search for faculty investigators to conduct the study was handled through issuing 
a request for applications (RFA) to all 20 CSU campuses. The response to this RFA by 
CSU faculty was low with several possible reasons, e.g., inadequate funding for the 
task; ineligibility to apply by former CSU assessment project grantees; insufficient 
distribution of the RFA on campuses; lack of faculty interest due to non-traditional 
area of research. A faculty research team at CSU San Bernardino, however, was 
identified and recommended by the steering committee to conduct the study. 

• The study was difficult to design because the pilot projects under investigation varied 
widely in their research questions, methodological approach, data generated, overall 
project environment, and level of sophistication. The study as designed was 
qualitative rather than quantitative. 

• The results of the study showed that four major factors affect the success of 
implementing assessment programs: (1) adequacy of assessment measures 
developed/adopted; (2) existence of administrative support; (3) level of faculty 
participation by all members responsible for implementation of any aspects of 
assessment programs; and (4) project director’s training and experience in assessment 
procedures. The relationships of numerous variables to successful 
implementation of assessment were analyzed and reported. 

8 
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• Two issues of "Assessment in the CSU" newsletter insert were published: (1) in 
the fall of 1990 which contained the plan for the FIPSE-sponsored study along 
with highlights of campus assessment activities; (2) in the spring of 1991 which 
contained preliminary findings of the study and the announcement of the CSU 
assessment seminar in June. Both issues were distributed to all 22,000 full time 
and part time faculty and academic administrators in the CSU system and to 
approximately 2,000 other higher education institutions and organizations 
nationwide. 

• Dissemination of the study results and other information on assessment began 
early in the project and still continues. The activities included: (1) presentation 
of the evaluation results by the research team and presentations of numerous 
CSU campus pilot programs by project directors at a CSU systemwide 
assessment seminar, in conjunction with the American Association for Higher 
Education (AAHE) national assessment forum, which brought about 100 CSU 
faculty and participants from other higher learning institutions; (2) 
presentations of several CSU panels at various national meetings, including the 
1991 AAHE assessment forum, the 1991 the Society for College and University 
Planning (SCUP) conference, and the 1992 American Educational Research 
Association (AERA) meeting; (3) discussions of the CSU-FIPSE project occurred 
at the 1991 Assessment Institute and at the 1992 AAHE assessment forum; (4) 
publication in June 1992 and continuous dissemination of a monograph on 
student outcomes assessment which contains chapters on the evaluation 
results, campus case studies, and the development of CSU assessment policy; 
(5) a companion videotape production which contains interviews with 
assessment researchers and practitioners discussing strategies for 
implementing assessment programs; (6) a booklet on student outcomes 
assessment which summarizes the 15 CSU assessment pilot projects and the 
results of the FIPSE evaluation study to be published and disseminated in 
spring 1993 under the CSU Academic Program Improvement "Academic 
Challenges" Series. 

Project Results 

What Faculty Learned from the Project 

• Faculty who were involved in the study or participated in the dissemination 
activities learned the following elements: (1) important factors that affect the 
success of implementing assessment programs on campus; (2) national 
perspectives on student outcomes assessment; (3) CSU experiences, practices, 
and policy in outcomes assessment; and (4) the continuous momentum of 
integrating assessment into teaching practice and using assessment results for 
educational improvement, including discussions of accreditation guidelines 
related to assessment. 
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Evaluation of the Project 



• The project was monitored closely by the project director to assure the timelines 
and deadlines associated with the study and dissemination activities were met. 
Revisions of the timelines have been made along the way to accommodate 
fluctuations. 

• Estimated 200 faculty and administrators across the nation were directly 
involved in research or dissemination activities associated with this project. 
Another 500 received the project results and products through the mail. All 
22,000 CSU faculty and academic administrators received two issues of the 
newsletter “Assessment in the CSU." 

• The study results have been widely disseminated throughout the nation, 
including presentations at the CSU systemwide seminar and national meetings. 
The participants at the CSU assessment seminar evaluated the meeting 
positively and pointed out that the most valuable aspect of the seminar was the 
opportunity to meet colleagues and exchange experiences. The study has thus 
far received very good recognition and visibility. 

Plans for Continuation and Dissemination 

• The two major end products of this study, a monograph on assessment 
practices and experiences in the CSU and a videotape featuring interviews with 
assessment researchers and practitioners on the topics of effective 
implementation strategies, will continue to be made available nationally 
through CSU's Academic Publications Program. 

• A booklet containing summaries of the campus pilot projects and results of this 
evaluation study will be published by the CSU in 1993. This booklet will be 
included in the CSU Academic Program Improvement "Academic Challenges" 
series and disseminated through the CSU Academic Publications Program to 
interested parties in the state and nation. 

• Further dissemination of the study results and other CSU assessment activities 
will continue through programs administered by the Institute for Teaching and 
Learning, such as ITUs annual national conference. The Teaching and Learning 

Exchange. 

• The Institute for Teaching and Learning will continue to work closely with 
other higher education constituencies, such as the National Center for 
Teaching, Learning, and Assessment and the Western Association of Schools 
and Colleges (WASC), to make assessment an integral part of regular 
instructional practices and to improve education by using assessment results. 
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Summary and Conclusions 



• Convening a national steering committee to guide the development of the 
project caused scheduling problems and some project delays, but the improved 
project design and products that resulted from the committee's national 
perspectives made this aspect of project management worth the trouble. 

• Faculty and administrators interested in assessment need and appreciate 
opportunities to meet colleagues to keep the assessment dialogue open, to 
exchange their experiences, and to make the assessment momentum continue 
in higher education. This is especially important during times of budget 
cutbacks. 

• Our experience with this project confirms that, outside of a few selected 
disciplines, faculty often lack interest in conducting research in student 
outcomes assessment. To begin to turn this around, appropriate rewards must 
be provided for faculty who engage in such research. 

• Faculty who participated in this project indicated a desire for more step-by-step 
instructions on implementing assessment in their classrooms and departments. 
In planning for future dissemination or technical assistance activities, we will 
include: (1) presentations by experienced faculty who have actually 
implemented assessment programs; (2) more training on the development and 
use of assessment instruments; (3) small group working sessions on specific 
problems; and (4) more examples of successful assessment in subject matter 
areas. 
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CSU Receives FIPSE Grant 

■ ■ ■ The CSU has been awarded a 
grant from the United States Department of 
Education Fund for the Improvement of 
Postsecondary Education to conduct an 
evaluation study to determine those factors 
contributing to the relative success of the 15 
CSU student outcomes assessment pilot 
projects funded by the Academic Program 
Improvement Campus Grants Program. 
According to Dean Helen Roberts, “Results 
of this study are of potential national 
importance in understanding how to 
implement student outcomes assessment 
at large public universities. We want to 
share our findings and disseminate validated 
assessment practices to other faculties and 
institutions." 

A blue ribbon steering committee was 
appointed by Lee Kerschner, vice chancellor 
for academic affairs, to guide the project. At 
the committee’s initial meeting held in 
January, a number of recommendations 
regarding project design and implementation 
were proposed and issues regarding the 
dissemination of findings were discussed. 
Members of the steering committee include: 
Bernard Goldstein, professor of biology and 
director of research and professional 
development, San Francisco State 
University; Becky Loewy, professor of 
psychology and vice chair of the statewide 
Academic Senate, San Francisco State 
University; Priscilla Chaffe-Stengel, 



professor of information systems and 
decision sciences, CSU Fresno; Olita Harris, 
professor of social work, San Diego State 
University; John Toffel, director, Program 
Excellence Initiatives, Ohio Board of 
Regents; Peter Ewell, senior associate, 
National Center for Higher Educational 
Management Systems; Daniel Stufflebeam, 
director, Evaluation Center, Western 
Michigan University; Frank Young, associate 
dean of academic affairs for plans, Office of 
the Chancellor; and Angel Sanchez, 
associate director of analytic studies, Office 
of the Chancellor. 

Duri ng the second year of the grant, findings 
of the study will be published by the CSU 
Institute for Teaching and Learning and 
disseminated broadly within California and 
nationally. By the fall of 1 990, the CSU will 
have in place a systemwide policy statement 
on student outcomes assessment. The 
results of this study are of critical importance 
to CSU campuses as they move to 
implement the new policy and to comply 
with new guidelines of the Western 
Association of Schools and Colleges. 
Additional dissemination activities will 
include issuing a series of newsletters to 
CSU faculty, and a summer workshop for 
faculty who are planning to implement 
assessment programs in their departments 
or campuswide. 



Hayward Faculty to Study 
Writing Assessment 

■ ■ ■ The Academic Program Improve- 
ment Campus Grants Prog ram funded three 
projects related to student outcomes as- 
sessment during the 1989-90 grant cycle. 
Mary Cullinan, English Department, CSU 
Hayward, was one such recipient for her 
proposal entitled “Assessment of Student 
Outcomes: A Basic Writer’s Writing Pro- 
gram." Cullinan states, “Our goal for the 
1990’s is to insure that our basic writing 
program addresses the needs of all stu- 
dents who are at risk in the university com- 
munity because of language impediments. 
This program assessment will help us to 
examine the factors that help students write 
well; it will also enable us to modify the 
program so it is more sensitive to the prob- 
lems students experience as they face the 
pages of their writing assignments.” 

Assessment of student writing will focus on 
a holistic evaluation of student portfolios 
and essays. Other evaluative procedures 
will include perceived writing improvement, 
identification 
of activities 
that increase 
effective writ- 
ing, compari- 
son of reme- 
dial level to 
•progress, 
and follow-up 
of freshman 
composition 
experience. 

Cullinan goes 
on to say that Mary Cullinan 

the wide variety of data acquired will con- 
tribute to the growing fund of information 
about the teaching of writing to native and 
non-native basic writers. “Our findings 
should be extremely useful to administra- 
tors and faculty on other campuses where 
basic writing classes are becoming an in- 
creasingly important part of the curriculum." 



Commentary 

■ ■ ■ The Institute for Teaching and 
Learning invites all CSU faculty, 
administrators and professional staff to 
submit commentary on assessment issues 
and practices for publication in this column. 
Topics may range from the theoretical and 
philosophical to the more pragmatic 
QYnonances associated with implementation 
m > ampus and department levels. This 

tKJC 



is an opportunity to share with your 
colleagues information on what has been 
happening at your campus and what insights 
you’ve gleaned. Send copy to Pamela 
Krochalk, Assistant Dean, Institute for 
Teaching and Learning, 400 Golden Shore, 
Long Beach, CA 90802-4275; FAX (213) 
590-5749. 
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Marylu Mattson Heads Liberal Studies 
Assessment Project at Sonoma 



■ ■ ■ Professor Marylu Mattson, 

Hutchins Schoolof Liberal Studies, Sonoma 
State University, suggests that although 
faculty may believe that assessment of 
students’ coursework provides some 
indication of program effectiveness, it is of 
little value to the individual student who is 
being assessed. 

Through funding from the Academic 
Program Improvement Campus Grants 
Program, Mattson says she is attempting to 
overcomethisshortcoming inherent in many 
“outcomes assessment" plans. Faculty in 
liberal studies at SSU will design a system 
wherein majors develop an individualized 
learning plan and a portfolio demonstrating 
progress towards the goals specified in the 
plan. A junior-level “gateway course" will, 
among other things, allow adequate time for 
onefaculty memberto evaluatethe student's 
strengths and weaknesses in several areas, 
introducethestudenttothe portfolio process, 



and help the student establish a personal 
study plan for the major. Advising in 
subsequent semesters will use the 
developing portfolio as an ongoing means 
of helping student and advisor understand 
what progress is being made toward the 
goals established at the outset for the 
individual student. During a capstone 
course, an assessment will be made of the 
portfolio as well as the development of 
higher order skills. 

The goal of the project is to integrate 
formative and summative assessment into 
the curriculum in a way that will aid the 
student’s development as a self-motivated 
learner. During the first phase of the grant, 
five faculty taskforces will examine various 
aspects of the curriculum and design a 
processforthe inclusion of portfolios. During 
the second phase of the pilot project, 
portfolios will be incorporated in a course 
introductory to the major. 



Nursing Ties Assessment 
to Job Competencies 
at Chico 

■ ■ ■ Bessie Marquis, professor of 

nursing, CSU Chico, was funded through 
the Academic Program Improvement 
Campus Grants Program to conduct “Out- 
comes Assessment of Four Classes of Nurs- 
ing Graduates." According to Marquis, the 

purpose of 
the study is to 
determinethe 
extent to 
which four 
consecutive 
classes of 
graduated 
nursing stu- 
dents have 
achieved and 
continue to 
demonstrate 
on-the-job 
competen- 
cies deline- 
ated by the 
faculty as the end-of-program objectives. 
Based on a 1982 study, the current project 
is expanded to include further analysis of 
existing data, extent to which graduates 
meet the nursing practice competencies 
specified in the curriculum, relationships 
between existing measures of student 
competence and subsequent success in 
the workplace, and comparisons of alumni 
assessments overtime. The results of the 
study will be used to make appropriate 
changes in curriculum design and imple- 
^ ‘ation. 



CSU Report on Student 
Outcomes Assessment 
Ready for Approval 

■ ■ ■ The draft report of the CSU Ad- 
visory Committee on Student Outcomes 
Assessment has been completed and will 
be taken before the statewide Academic 
Senate for action in March and the Board of 
Trustees for final approval in May. 

The committee was established in Novem- 
ber 1987 by Chancellor Reynolds who 
charged its members with studying student 
outcomes assessment and advising the 
Chancellor on related policies; coordinating 
responses to the California Postsecondary 
Education Commission in connection with 
its study of outcomes assessment; and 
submitting a report and recommendations 
for directions the CSU should take with 
regard to outcomes assessment. 



Subscribe Now 

■ ■ ■ Stay current on trends and prac- 
tices in assessment in higher education. 
Subscribe to Assessment Update , a quar- 
terly newsletter published by the Center for 
Assessment Research and Development, 
University of Tennessee, Knoxville. Re- 
quests for subscriptions should be sent to 
Jossey - Bass, 350 Sansome St., San Fran- 
cisco, CA94104-1310. Tel.: (415) 433-1 766. 
Rates: $60 per year and 20% discount if 
prepaying five or more subscriptions. 
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San Diego Sponsors 
Conference on Student 
Outcomes Assessment 

■ ■ ■ San Diego State University plans 
a campuswide conference on student out- 
comes assessment entitled “Assessment of 
Learning: Who, What, When, Why & How." 
According to Olita Harris, chair of the Uni- 
versity Assessment Committee, which 
sponsors the conference, “The purposes of 
the conference are to bring togetherfaculty, 
administrators, and students to share infor- 
mation about assessment activities under- 
taken by the various departments and to 
create a widening base for assessment 
activities, expertise, and enthusiasm on this 
campus." Scheduled for February 23, the 
conference is designed to offer participants 
the opportunity to voice their apprehen- 
sions, frustrations, successes and ideas in 
a collegial atmosphere. Topics to be dis- 
cussed include statewide initiatives, policy 
environment for assessment of learning, 
assessment of learning in the disciplines, 
assessment models, standardized instru- 
ments, and instrument development and 
use. 



■ Meetings and Events ■ 

March 26-27: The 1990 ACT National 
Assessment Conference, “Accreditation and 
Accountability -Challenges for Assessment," 
Kansas City, MO. Contact: Donna 

Appleglise, ACT Educational Services 
Division, P.O. Box 168, Iowa City, IA 52243. 
Tel.: (319)337-1032. 

June 27-30: Fifth National Conference on 
Assessment, The Washington Hilton in 
Washington, D.C. Last call for papers -due 
immediately. Contact: Barbara D. Wright, 
Assessment Forum, American Association 
of Higher Education, One DuPont Circle, 
Suite 600, Washington, D.C. 20036. Tel.: 
(202) 293-6440, FAX: (202) 293-0073. 

October 19-21: Annual Meeting of the 
American Evaluation Association, San 
Francisco. Technical Interest Group on 
Assessment in Higher Education has sched- 
uled five sessions. Contact: Mary Anne 
Bunda, Director, Office of University As- 
sessment, Western Michigan University, 
Kalamazoo, Ml 49008-5130. Tel.: (616) 
387-3031. 

October 19-21 : Third Colloquium on Writ- 
ing Assessment, Missouri Western State 
College. Contact: Renee Betz, Central 
Missouri State University, Warrensburg, MO 
64093. Tel.: (816)429-4780. 
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FIPSE- Funded Assessment Study Nears Completion 



■ ■ ■ A California State University study 
sponsored by the Fundforthe Improvement 
of Postsecondary Education (FIPSE) is 
identifying factors which contribute to 
effective assessment programs. The study, 
conducted by psychologists Matt Riggs and 
Joanna Worthley of CSU San Bernardino, 
is designed to evaluate the effectiveness of 
1 5 CSU student outcomes assessment pilot 
projects funded in 1 986-90 by the Academic 
Program Improvement (API) Grants 
Program. Findings of the study are due to 
be released at a statewide conference on 
June 8, 1991. 

Data collection for the evaluation began in 
fall 1 990, with an extensive survey mailed to 
pilot project directors. Discussion of as- 
sessment activities and project outcomes 
continued at a meeting of API project di- 
rectors with the evaluation team in November 
at San Francisco State University. The San 
Francisco meeting allowed a productive 
exchange among faculty whose experience 
with the pilot assessment initiatives is ex- 
pected to provide direction for continuing 
assessment in the CSU," according to 
Dr. Riggs. Data were also collected from 
several other sources, including: 1)afo!low- 
up survey focused on multicultural dimen- 
sions of the pilot projects; 2) API project 
directors’ final reports; and 3) external 
evaluation reports of the API projects. 

“Evaluation of the relative effectiveness of 
the pilot projects involves a careful de- 
scription of the relationship of assessment 
environment and methods to assessment 
outcomes," said Dr. Worthley. To demon- 
strate the relationships among these factors 
for each project, the researchers developed 
a “site-ordered predictor-outcome matrix." 
The matrix displays a project’s standing on 
both environmental variables (such as fac- 
ulty ownership, administrative support, and 
campus attitudes toward assessment ac- 
tivities) and methodological variables (such 
'Q definition of project goals, attention to 

ERK — 



measurement properties of proposed in- 
struments, and multi-measurement ap- 
proaches to assessment). This matrix, to- 
gether with a large database of qualitative 
information on the projects, is expected to 
yield two important indices of effectiveness: 
1) a “checklist" of environmental and 
methodological factors, in order of their 
importance, which contribute to effective 
assessment; and 2) an estimate of the 
replicability of the various measurement 
strategies used in the pilot projects, based 
on the availability of critical factors in a 
proposed assessmentsetting. Data analysis 
will be focused on describing the “goodness 
of fit" between project characteristics and 
an extensive list of predictor variables. 

Members of the steering committee for the 
study are: 

Priscilla Chaffe-Stengel, Professor of 
Information Systems and Decision Sciences, 
CSU Fresno; 

Peter Ewell, Senior Associate, National 
Center for Higher Education Management 
Systems; 

Bernard Goldstein, Acting Director, 
Research and Professional Development, 
San Francisco State University; 

Olita Harris, Professor of Social Work, San 
Diego State University; 

Becky Loewy, Professorof Psychology, San 
Francisco State University; 

Angel Sanchez, Associate Director, Analytic 
Studies, CSU Office of the Chancellor; 
Daniel Stufflebeam, Director, Evaluation 
Center, Western Michigan University; 

John Tafel, Director of Authorization and 
Director of Program Excellence Initiatives, 
Ohio Board of Regents; 

Frank Young, Director, California Academic 
Partnership Program, CSU Office of the 
Chancellor; and 

Helen Roberts, State University Dean, 
Institute for Teaching and Learning, CSU 
Office of the Chancellor. 
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Systemwide Assessment 
Seminar Scheduled 
in June 1991 

■ ■ ■ The CSU Institute for Teaching and 
Learning issponsoring a systemwide faculty 
seminar on student outcomes assessment, 
scheduled for Saturday, June 8, 1 991 , from 
10:00 am - 6:00 pm. The CSU seminar, 
entitled “Toward a New Paradigm: As- 
sessment in Multicultural California," will be 
held in conjunction with the American As- 
sociation for Higher Education Conference 
on Assessment in Higher Education, both at 
the San Francisco Hilton Hotel. 

Results of a statewide evaluation of fifteen 
CSU assessment pilot projects will be 
released at the seminar. Discussions of 
these results and other assessment issues 
related to program and institutional 
effectiveness in our multicultural university 
will be on the agenda. The seminar will also 
feature presentations by the project directors 
of several successful CSU assessment 
projects funded by Academic Program 
Improvement (API) grants, along with 
comments by the following distinguished 
speakers: 

• Peter Ewell, Senior Associate, National 
Centerfor Higher Education Management 
Systems; 

• Sherrin Marshall, Program Officer, Fund 
for the Improvement of Postsecondary 
Education; 

• Ralph Wolfe, Associate Director, Western 
Association of Schools and Colleges. 

Each CSU campus has been invited to 
select a team of people to attend. For more 
information, contact Dr.Olita Harris, Seminar 
Chair and Coordinator, School of Social 
Work, San Diego State University, (619) 
594-6860. 

This seminar isbeing made possible through 
a grant from the Fund for the Improvement 
of Postsecondary Education (FIPSE). 
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Resource Guides Developed for Teacher Assessment 



■ ■ ■ The CSU Board of Trustees 

adopted a policy in September 1985 
calling for faculty in the academic 
disciplines to assess and certify the subject 
matter competence of prospective 
teachers. Systemwide workgroups have 
been formed to develop resource guides 



for assessment in various disciplines. 
Faculty participating in CSU subject matter 
assessment development are also 
contributing to state and national 
development of improved teacher 
assessment processes. Resource guides 
have now been completed in eight 



disciplines by each coordinating campus: 
art (Los Angeles), English (systemwide), 
foreign language (San Diego), liberal 
studies (systemwide), life science 
(Sacramento), music (San Francisco), 
physical education (Fresno), and social 
science (San Diego). 



CSU Faculty Present Assessment Outcomes at National Meetings 

■ ■ ■ Several groups of CSU faculty and administrators are presenting CSU experiences and research findings in student 
outcomes assessment at national professional meetings during 1991 . 
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EVALUATION OE 



STUDENT OUTCOMES 



ASSESSMENT E I LOT UR.OU EOT S 

IN THE 

CALIFORNIA STATE UNIVERSITY 



EXECUTIVE SUMMARY 

Beginning in 1985, Academic Program Improvement grant funds 
have supported a series of student outcomes assessment projects 
initiated by faculty in a variety of disciplines on 15 campuses 
of the CSU . These pilot projects were aimed at demonstrating how 
student outcomes of various kinds can be used to assess the 
effectiveness of General Education and baccalaureate degree 
programs . 

The present study, sponsored by the Fund for the Improvement 
of Postsecondary Education (FIPSE) provides a ”meta-assessment " 
of the 15 pilot projects. The purpose was to define features 
which were consistently associated with project effectiveness. 



METHOD 

A three-part framework was used to specify variables and to 
organize the data. Relevant variables were conceptualized within 
the categories of environmental factors, methodological factors, 
and assessment project outcomes (see Figure 1). Data were 
obtained from project final reports, directly from the project 
directors (via mail surveys, phone surveys, and personal 
contact), other reports or articles resulting from each 
assessment project, and the reports submitted fey the projects 
external evaluators. 
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FIGURE 1: Conceptual Model 
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ANALYSIS 



All data were condensed and entered onto a "me ta-ma r r ix . ” 
This master chart contained information relevant to ail thirty- 
seven variables for ail projects included in the final 
assessment. . On the basis of information represented in this 
matrix, a qualitative categorization of all projects on ail 
variables was completed. For each project, all variables were 
classified as: 

4 - strongly present/ achieved, 

3 - partly present/achieved, 

2 - weakly present/achieved, 

1 - absent/not achieved. 

Environmental and methodological factors served as predictor 
variables. Criterion scores were produced using: 1) the average 

of all outcomes, 2) the average of outcomes categorized as direct 
outcomes, and 3) the average of indirect outcomes. All scores 
were then standard i zed . 

"Consistency" between predictors and outcomes was assessed 
by computing the squared deviations between each predictor and 
the average "outcome (overall, direct, & indirect), summing 
these squared deviations across project sites, and dividing the 
resulting value by the number of projects. The resulting 
"variance with outcomes" across sites is smallest for those 
variables that were most consistently related to outcomes. 

These results are summarized in Table 1. The rank orders 
enable comparison of the relative importance of predictors. 
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Table 1 : 



Predictor Variable Rank Orders 
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RESULTS 



ENVIRONMENTAL FACTORS 
Faculty Involvement Variables 

Two faculty variables were consistently toward the top of 
the respective rank ordering’s. "Faculty Participation and 
"Perceived Faculty Workload" appear to be important indicators of 
assess me nt success. 

"Faculty Ownership" and "Faculty Consensus with Plan" were 
also relatively good for direct outcomes. "Faculty Participation 
in Planning" appears to have little to do w.ith project outcomes. 
Training/Experience with Student Outcomes Assessment 

Project outcomes were associated with the project director s 
efficacy in outcomes assessment. Faculty training and the 
project director’s general academic background and experience 
were not as closely tied to project results. 

Support Variables 

Administrative support was closely tied to project outcomes, 
both direct and indirect. Neither adequacy of budget, supplies, 
and other institutional resources nor student support were 
closely associated with project outcomes. 

Existing Student Outcomes Assessment Procedures 

Prior experience with assessment did not seem to have a 
consistent positive or negative effect on outcomes. 

Project Focus 

The content area in which the project occurred had little to 
do with the relative success of the project. However , the 
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intended breadth of the audience was very closely associated with 
outcomes. The "Breadth of Audience" construct was anchored at 
the low end by those projects reporting only to the funding 
institution and at the high end by projects disseminating their 
results nationally. 

METHODOLOGICAL FACTORS 
General Procedural Adequacy 

The single most critical variable from this category was the 
development/adoption of good measures of student outcomes. 
Appropriate selection of outcomes to measure was moderately 
associated with project outcomes. 

"Project Goal Definition," the adequacy of "Statistical 
Analyses" conducted, "Data Cc 1 lec t ion/Repo r t ing , " and' the 
reporting of "Measurement Properties" did not appear to co-vary 
tightly with project results. 

Project Comprehensiveness 

The "Development of Multiple Measures" of student outcomes 
and "Sensitivity to Multicultural Issues" appeared moderately 
associated with project outcomes. The comprehensiveness of the 
reports made available were not closely related to project 
outcomes . 

Cost Effectiveness of Project 

The "Utility/Economy” of the projects’ assessment procedures 
was near the bottom of the rank ordering. Apparently expensive 
projects in terms of dollars spent to students assessed w _ ere not 
always the richest in results. 
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SUMMARY 



Though qualitative analysis does net enable the same level 
of precision as might be obtained from more quantitative data, 
the observations and resulting classifications produced in this 
study were systematic, relatively objective, and almost always 
based on multiple sources. Additional testing and replications 
of this study’s conclusions is suggested, and could occur within 
the CSU’s continuing program to develop and monitor programs of 
student outcomes assessment. 

The results of this analysis suggest that future initiatives 
be especially sensitive to four variables: 1) the adequacy of 

the measures developed/adopted , 2) the development of 

administrative support, 3) faculty participation by all members 
responsible for implementation of any aspect of the assessment 
project, and 4) the assessment project director s 

training /experience with student outcomes assessment proceduie^. 
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INTRODUCTION 



The current assessment movement in higher education is 
driven by the wary partnership of reform and accountability, a 
partnership yielding a complex and diverse collection of 
assessment activities in university settings (Ewell, 1991). Over 
the last several years, the California State University System 
has moved to construct an assessment agenda which responds to 
both reform and accountability in ways that will preserve the 
commitment of the CSU to intellectual and programmatic diversity. 
Beginning in 1986, Academic Program Improvement grant funds have 
supported a series of student outcomes assessment projects 
initiated by faculty in a variety of disciplines on 11 campuses 
of the CSU. These pilot projects, under the aegis of the CSU 
Institute for Teaching and Learning were aimed at demonstrating 
how student outcomes of various kinds can be used to assess the 
effectiveness of General Education and baccalaureate degree 
programs . 

The present study, sponsored by the Fund for the Improvement 
of Post secondary Education (FIPSE) provides a "meta-assessment" 
of the 15 pilot projects. These projects, briefly detailed in 
Table 1, have developed and field-tested a variety of assessment 
measures/ ins truments , including portfolios, interviews, 
senior/capstone projects, surveys, and examinations. (For a 
fuller description of projects with their data sources, see 
Appendix A ) . 
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Table 1 



CALIFORNIA STATE UNIVERSI 
ASSESSMENT 


TV FI LOT PROJECTS IN STUDENT OUTCOMES 


Director & Came us 


Proieet Title L Focus 


Betty Biecha 
Leigh Mint 2 
CSU Hayward 
with 

Newman Fisher 
Richard Giardina 
San Francisco S U 
with 

Leon Dorosz 
Howard S he 1 1 hammer 
San Jose S U 


Assessment of Majors: A Three-Campus, 

Three-Discipline Model 
Focus: Development of comprehensive 

examinations for seniors in 
biology, economics, and mathematics 


Priscilla Chaffe-Stengel 
CSU Fresno 


Assessment of Undergraduate 
Reading Competence 

Focus: Assessment of student reading 

strategies and competence related 
to course assignments and library 
skills 


Priscilla Chaffe-Stengel 
CSU Fresno 


Assessment of Undergraduate 
Writing Competence 
Focus: Assessment of student 

performance on the Upper Division 
Writing Exam as a function of 
course exposure and language 
prof iciency 


S . Eugene Clark 
CSU Bakersfield 


Knowledge and Attitudes in General 

Education: A CSU-Communi ty College 
Joint Assessment 

Focus: Assessment of impact of GE 

course in Western Civilization on 
students’ knowledge and values 


P. Chris Cozbv 
Jeffry Young 
CSU Fullerton 


Student Outcomes Related to Curricular 
Variety in Gerontology 
Focus: Development of a model for 

cross-campus assessment of outcomes 
for interdisciplinary programs in 
gerontology in the CSU 
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Director fc Cammis 


Pro iect title & Focus 


Mary Cull i nan 
CSU Hayward 


Assessment of Student Outcomes: A Basic 

Writer’s Writing Program 
Focus: Development of a model for 

assessing outcomes in the Intensive 
Learning Experience (ILE) writing 
course sequence 


Catharine Dezseran 
Peter Grego 

CSU North ridge 


Student Outcomes Assessment in Academic 
Program Improvement in Theatre 
Focus: Development of a performance 

-based mastery test for summative 
and formative assessment of student 
achievement in theatre 


Catharine Lucas 

San Francisco S U 


Assessing Outcomes for English Teacher 
Candidates 

Focus: Development of an assessment 

course” to evaluate the subject 
-matter competency of teacher 
credential candidates in English 
language arts 


Bessie Marquis 
CSU Chico 


Outcomes Assessment of Four Classes of 
Nursing Graduates 

Focus: Development of a mult i -measure 

assessment of nursing program 
graduates to identify trends in 
program effectiveness from 1983 - 
present 


Marylu Mattson 
Sonoma S U 


Integrating . Student Outcomes Assessment 
into the Curriculum 
Focus: Development of a portfolio 

system to assess formative and 
summative outcomes for students in 
an interdisciplinary liberal 
studies program 


Andrew Moss 

C S P U, Pomona 


Enhancing Quality by Assessment: 

A General Education Project 
Focus: Development of a comprehensive 

assessment program for an 
Interdisciplinary General Education 
Program 



Director & Campus 



Project Title & Focus 



Kenneth L. Nyberg An Empirical Evaluation of Five 

CSU Bakersfield Baccalaureate Social Science 

Programs 

Focus: Development of a model to 

conduct longitudinal assessments of 
student performance and perceptions 
of degree programs in anthropology, 
economics, political science, 
psychology, and sociology 



Harry Polkinhorn 
San Diego S U 



Student Outcomes Assessment: Liberal 

Studies Major 

Focus: Development of a multi-measure 

assessment program for student 
outcomes in liberal studies 



This "meta-assessment" of the projects is designed to define 
features across the 15 projects which were consistently 
associated with effective assessment, and which might be expected 
to facilitate wider implementation of the assessment strategies 
field-tested experimentally. Moreover, because they represent a 
range of disciplinary perspectives, the pilot projects offer an 
opportunity to attempt a description of factors which predict 
assessment effectiveness across traditional boundaries. A 
summary of these features should contribute to the continued 
review of existing instructional approaches and administrative 
supports for the teaching/learning process in the CSL , as well as 
to the broader discussion of policy initiatives in assessment, in 
other university contexts. 

METHOD 

The project used a multi-site method with a mix of 
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qualitative and quantitative methods to evaluate factors that may 
have determined the outcomes achieved in fifteen California State 
University pilot assessment projects. A tares— part frame w or k was 
used to specify variables and to organize the data. Relevant 
variables were conceptualized within the categories of assessment 
environment, assessment methods, and assessment outcomes. 
Assessment En v i ronment 



The assessment environment wras defined as the social and 



organizational setting in which each project occurred. \ariables 
relevant to the determination of this construct included: 



1) general faculty participation in the planning of 

the project 

2) faculty participation in the impiemema ticn of the 

project 

3) the faculty’s perceived ’’ownership of the project 

( i.e. , self-determination) 

4) faculty consensus with the project plan 

5) faculty workload required by the project 

6) the project director’s experience in assessment 

activities 

7) the faculty’s experience in assessment activities 
8} the project director’s general academic experience 

9) adequacy of budget, supplies and resources 

10) administrative support 

11) student support 

12) previous experience with outcomes assessment 

13) the content domain (e.g., physical science) 

14) and the nature of the intended audience for the 

project’s results 



Assessment Methods 

Assessment methods were defined by variables that described 



the strategic aspects of each project- These variables are 
important to questions of replicability because they focus on the 
"goodness of fit" between specific procedural aspects of each 
project. Variables assessed included: 
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15) goal definition 

16) selection of appropriate outcomes 

17) the psychometric adequacy of measures used or 

developed 

IS) the success of data collection and reporting 
IS) the appropriate use of statistical analyses 

20) sensitivity to multicultural issues 

21) the use/de velopment of multiple outcome measures 

22) the comprehensiveness of reports describing the 

results of the project 

23) and the u t i 1 i ty/economy of procedures used 
Assessment Outcomes 



Outcome variables were used to capture the systematic growth 
and change that could be attributed to the assessment project. 

The following "direct" and "indirect" outcomes of the assessment 
projects were evaluated: 

DIRECT 

24) the survival of the project 

25) attainment of additional funding 

26) gains in student achievement 

27) curricular development 

28) improved teaching 

29) better student feedback 

30) the use of new' methods of assessment 

31) improved use of existing databases 

32) and the dissemination of results 

INDIRECT 

33) improved student recruitment/retention 

34) better general attitudes toward assessment 

35) new sources of money for assessment 

36) higher visibility of assessment 

37) and external adoption of measures or methods 

developed 

Conceptual Model 

The conceptual model guiding the analysis is based on the 
assumption that key "environmental and methodological 
variables will determine the nature of "outcome" variables. This 
model is illustrated in Figure 1. Included in each of the 



6 



.i 1 *-=. 



BEST COPY AVAILABLE 



categorical "bins” are the specific variables to be addressed. 
Data Collection 

Information relevant to the assessment of the variables 
described above was obtained from multiple sources. First, 
copies of all project reports were obtained. All reports were 
studied and assessed by both researchers conducting this analy- 
sis. Independent conclusions concerning the " presence/absence" 
or "adequacy/ inadequacy” of key variables were compared and 
evaluated. Where consensus on variable indicators could not be 
reached, that source of information was dropped from 
consideration. This process was especially important to the 
assessment of projects in reference to methodological variables 
(variables #15 to #23). 

The second, and perhaps most utilized, source of data was 
the project directors themselves. A survey was developed and 
administered to each of the project directors. A copy of the 
survey and a detailed report of the results of this survey are 
presented in a later section of the text. Results of this survey 
were especially critical in determining the nature of 
environmental variables for each project. These results were 
also primary determinants of the project outcome variables. 

Project directors were further utilized throughout the data 
collection period to fill in missing information and to provide 
updates on developments not available in the final project 
reports. A meeting with directors held in November at San 
Francisco State University yielded additional information on the 
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FIGURE 1: Conceptual Model 
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assessment environment on individual campuses. 



Telephone interviews with project directors continued i n w o 
the final weeks of this research. A special effort was made to 
obtain additional focus concerning multicultural issues relevant 
to outcomes assessment. A telephone interview was conducted. 
The results of this phone interview and additional information 
obtained from project files enabled an evaluation of the multicu 
ltural validity of measures used to assess student outcomes in 
the projects. The guidelines utilized for tnis analysis and 
major conclusions are reported later in this text. 

The final source of information utilized was the reports 
submitted by the external evaluators of the pilot projects. 

These were read last in an effort to maintain the cbjectiv it> of 
the researchers through the initial stages of the da^a 
collection. These reports were especially useful in 
supplementing observations concerning the methodological 
variables. 



RESULTS 

The data collection activities above provided three discrete 
steps in our report of the findings: 1) res u Its from the survey 

of project directors, 2) results from an assessment of 
multicultural validity in the projects, and 3) results from a 
summary assessment of generalizaoility of \ariables across 
project sites. These are reported separately in three sections 
following . 
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RESULTS FROM THE SURVEY OF PROJECT DIRECTORS 



This survey of pilot project directors was a preliminary 
data gathering step in a study designed to evaluate a series of 
student outcome assessment projects conducted in the California 
State University. Issues to be addressed by the final analysis 
which will utilize this data are: 1) the identification of 

general factors that contributed to the relative effectiveness of 
these fifteen CSU pilot projects, and 2} the assessment of 
potential for replicability of successful projects in other 
departments, CSU campuses, and colleges and universities across 
the nation. 

Though the wide variety of programs being studied prohibits 
simple across-the-board comparisons, this survey attempted to tap 
certain contextual and methodological variables that may 
generalize across institutions and content areas. General 
aspects of each project’s environment that were assessed by this 
survey included faculty ownership/commitment to the. project, 
training/exper ience of the project personnel, external support 
obtained for the project, and previous experiences with and/or 
attitudes toward student outcomes assessment. Information 
relevant to the general methodology of each project was also 
obtained. This included data relevant to the number and nature 
of outcomes assessed, methods of measurement, and the 
comprehensiveness of data collected. 

The survey also enabled project directors to describe the 
relative effectiveness of their projects in reference to such 
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variables as successful funding/continuation of assessment 
programs, successful development/use of outcome measures, 
improved faculty/student attitudes, improvements in curriculum 
and/or instruction, improved academic self-evaluation, and other 
indirect project outcomes. This data enabled the assessment of 
general relationships between contextual variables and subsequent 
project effectiveness. A copy of the survey instrument is 
attached in Appendix B. 

Data Summary 

Though fifteen projects are listed, one project involved 
discrete events at different locations within different 
departments. In the data description, these are treated as 
separate projects giving the analysis a total n of 1 i . There 
was only one duplication of responses for the same project. This 
duplication was adjusted for in variables requiring a count of 
representative projects. 

Survey items 1 and 2 asked directors to report the number of 
faculty that participated in each project. The number of faculty 
involved in the initial planning of each project ranged from 2 to 
100 with a mode of 4. The number of faculty involved in project 
implementation ranged from 2 to 400 (one project ended up 
university-wide in its second year ) with a mode of 14. 

Items 3-9 measured different aspects of faculty "ownership 
of the project. The data revealed a wide range of responses.. 
Though there were some glaring exceptions, directors generally 
reported high levels of faculty participation in the 
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implementation of the project, support for the subsequent goals 
of the project, and consensus with the project plan. Projects 
differed significantly, however, in reference to the inclusion of 
faculty participants in the planning/development of the project, 
faculty agreement with the nature of student performance 
criterion, perceived work loads resulting from the project, and 
perceptions of ownership among faculty. 

In response to the request for a description of any other 
factors related to faculty involvement and support, one director 
very simply summarized this point by reporting that "lack of 
staff support made implementation impossible." There was one 
report of faculty suspicion of the Chancellor’s Office concerning 
how the results of the study might be used. One director 
reported that to garner support, they first took their proposed 
project to several faculty committees, including their faculty 
senate. Before they had a chance to consult with their own 
faculty, the senate returned their project as a new program 
mandate . Some quick diplomacy was applied, and good support was 
ma inta i ned . 

Items 10 and 11 asked the director to rate the level of 
relevant training specific to student outcomes assessment for 
themselves and the project’s participating faculty. Directors 
generally rated themselves as fairly sophisticated. There was 
more variance in reference to their participating faculty. 

Items 12-14 asked for educational and experiential 
attributes of the project directors. All directors possessed 
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terminal degrees in their areas. In general, they were 
experienced teachers with an average of 16 years in the 
classroom. Most directors were full professors, and all but four 
were tenured. 

Items 12-14 asked for educational and experiential 
attributes of the participating faculty. Almost all held 
terminal degrees in their areas, a large majority were tenured, 
full professors. Directors also reported that participating 
faculty general had many years experience with an overall average 
of 17 years in the classroom. 

When asked to describe special training or experience that 
may have facilitated the project, about half of the directors 
reported a fair to extensive amount of background specific to 
student outcomes assessment. Others reported related specialized 
skills in measurement, student retention, and program evaluation. 

Most directors did not report extensive background for the 
other faculty participants, but several comments were made 
concerning how much was learned in the process of implementing 
the projects. 

Items 15-16 asked directors to describe sources of 
additional funding/support they received for their projects in 
the initial year. Ten directors reported receiving additional 
funds beyond the required matching funds from their university. 
The most frequent source of additional support was money from the 
AAC/FIPSE grant. Three projects received additional money from 
on- campus sources. 
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Item 17 measured the director’s perception of support for 
their project from the university’s administration. Directors 
generally felt this source of support was good. Responses ranged 
from "4" to ”7" with a mode of ”7." 

Items 18 and 19 measured the director’s perception of 
student support for the project. Directors reported that 
students supported the project (response range of "4'' to "7", 
with a modal response of ”6''), but that students offered 
significant resistance to changes associated with the 
implementation of the assessment (response range of '"l" to "6”, 
with a modal response of "1”). 

Items 20-24 determined whether or not. there were previously 
existing outcome assessment mechanisms/programs in place, and, if 
so, how these affected the project. Seven directors reported 
existing outcomes assessment. In item 21, four of these s e \ e n 
reported that the new program meshed well with existing programs. 
In item 22, four of the seven (not the same four) reported that 
the goals of their new project were consistent with the existing 
program’s goals. In item 23, two directors reported that it was 
more true than false that existing negative attitudes adversely 
affected the implementation of their project. In response to 
item 24, two directors reported that it was more true than false 
that familiarity with outcomes assessment facilitated the 
implementation of their projects. 

When asked in item 25 to describe in detail how any existing 
procedures or attitudes might have contributed to the 
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effectiveness of the pilot projects, responses were diverse. 

Five directors reported negative impact. Faculty suspicion and 
apathy were the major hurdles reported. One director reported 
that the faculty saw the project ”... as a threat at worst and a 
waste of time at best." 

Student suspicion and resentment of additional 
t e s t s / wo rkl oac were also reported. One director reported that 
faculty enthusiasm for assessment was the key to beating this 
obstacle; "...the average s tudent s . . . wi 1 1 cooperate in the 
development of alternatives for assessment if their faculty tell 
them it is important to do so.” 

There were four reports of positive impact. Faculty- 
education and exposure to successful student outcomes assessment 
procedures was reported to have facilitated two of the projects. 
An existing faculty desire to clarify program goals was reported 
as a contributing factor to the success of one project. 

In reference to the dissemination of their project reports 
(item 26), nine directors indicated that they had shared results 
beyond the reports requested by funding sources. Six projects 
have achieved nation-wide status via national organizations, 
conferences, or major publications. 

The institutional level at which projects were implemented 
filled the entire range from a single selectee class to a project 
that is now replicating the project at eighteen campuses 
nationwide. Altogether, three were applied to selected classes, 
five were department-wide, two were at the school le^el, three 
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were university-wide , and three were at least system-wide. 

Project goals (item 28) were as diverse as the programs they 
were intended to serve, and are consequently difficult to 
summarize. Many included the objective of testing the 
feasibility of outcomes assessment within the specific program 
type. Another common goal was the assessment of different types 
of external evaluators. The most common objective not directly 
related to the assessment process itself was the desire to 
clarify /develop programmatic objectives. 

The types of outcomes assessed (item 29) also varied 
dramatically. They included: assessment of simple content 

knowledge; demonstration of specific process skills; student, 
alumni, faculty, and the public’s attitudes toward the program; 
attainment of post-graduate goals (e.g., employment status, 
general satisfaction with preparation, employers’ satisfaction 
with the program’s graduates); and the development of specified 
attitudes/beliefs. 

Many methods of outcome measurement were employed (item 30). 
These included written examinations (objective and essay), oral 
■examinations, personal interviews, graded assignments, project 
evaluations, and attitudinal rating scales/surveys. The projects 
varied somewhat in how many methods of evaluation were used by 
each project. At least four used only a single type of measure. 

Items 32-33 were used to assess the type of demographic 
information that was collected as part of the projects data 
sets. Seven projects did not collect demographic information. 
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Three studies collected at least two demographic variables, and 
seven collected three or more. 

Item 39 asked if a per student or per class cost was 
estimated for the project. No project reported the assessment or 
estimation of any such value. 

Item 40 was used to determine how many projects have 
continued beyond the original year of funding. Seven projects 
have not been continued. Lack of funding was the main reason 
given for program closure. Ten directors reported that their 
project had continued (item 41). Of these, four had obtained no 
additional funding, and three were receiving very minimal 
departmental /un ive rs i ty support. Of the three who reported 
receiving additional support', one was an API continuation, one 
was an AAC/FIPSE grant originating before the API grant, and one 
did not report its source. 

Items 42-59 asked directors to assess the level of 
achievement of different potential project outcomes. Of those 
outcomes identified as direct outcomes (items 42-50), almost all 
directors reported development of good measures of student 
outcomes as an achievement of their projects. Most reported 
curricular improvements, increases in student feedback, 
clarification of instructional goals/objectives, increases in 
faculty assessment skills, and successful dissemination of 
information to other department s/ schoo ls/uni vers it ie s . As a 
whole, directors were less optimistic about gains in student 
achievement, improvement in teaching by the faculty involved, and 
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new or improved uses of existing databases. 

Of those outcomes identified as indirect outcomes (items 51- 
59), most directors reported improved self-evaluation of the 
academic program, improved faculty attitudes toward assessment 
activities, and greater visibility of assessment activities. 
Overall, somewhat less impact was perceived on student 
recruitment, student attitudes toward assessment activities, and 
institutional attitudes toward assessment. Directors reported 
less success with the development of new sources of 
revenue/support. Seven directors reported significant 
achievements in regards to the adoption of methods developed br- 
other department or institutions. 

When asked to describe other significant outcomes (item 60), 
four directors reported a significant deal of " se 1 f- improvement ’ 
in reference to their personal understanding of student outcomes 
assessment, their ability to teach, and clarification of their 
program’s objectives. Two also referred to similar improvements 
enjoyed by all faculty participating in the project. Two 
directors reported that the project had resulted in significant 
changes/improvements in their programs’ curriculum and definition 
of objectives. One director reported that the project formed the 
basis for a prototype assessment procedure that is being piloted 
for use statewide. On the negative side, one director s final 
observation was that the project ” . . . highl ighted the incredible 
apathy of most CSU faculty regarding the utility of outcomes 
assessment . " 
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Summary 

Survey responses indicate a good deal of variance in what we 
referred to as ’’faculty ownership” of the project. Some 
directors described high levels of faculty suspicion. Given that 
three of these variables were highly predictive of project 
outcomes and that faculty participation and support is generally 
acknowledged as a critical link to successful outcomes 
assessment, it will be important to further investigate how 
faculty involvement and support can be successfully achieved . 

Responses indicate that, on the. average, this group of 
project directors was a very experienced group, both in years at 
the university and in special training relevant to outcomes 
assessment. This may appear to present a problem in that it 
would be difficult to match the sophistication of these directors 
in future waves of outcomes assessment initiatives within this 
system; however, data collected here did not indicate that such 
experienced was a prerequisite to project outcomes. 

Obtainment of additional funding for the original projects 
and for continuations of the projects was somewhat sparse. It 
goes without saying that this will not soon improve. Funding was 
clearly described as a major hurdle tc further development of 
assessment programs. Somewhat discouraging was the fact that 
success in achieving project outcomes was not related to 
subsequent success in obtaining continuations of funding. 

Significant levels of student resistance were reported. 
Faculty promotion of the program was suggested as a key to 



48 



19 



diluting this problem. Given the relationship between student 
support and direct outcomes, specific recommendat ions/techniques 
for introducing assessment programs to students and maintaining 
student support should be sought. 

Five directors reported adverse impact due to faculty 
experience with pre-existing programs of outcomes assessment. 

Four described positive effects resulting from experiences with 
existing programs. Since it appears that pre-existing negative 
attitudes toward outcome assessment can be fatal to a new 
project, further efforts should be made to determine how programs 
can develop a positive atmosphere around existing assessment 
procedures . 

Several projects have been very successful in the 
dissemination of their results. Recognition and visibility 
should improve attitudes toward future outcomes assessment 
programs. 

Given current, financial circumstances, it may be unfortunate 
that no cost estimates were produced. Directors were not asked 
to compute such a figure, but meaningful start-up and maintenance 
costs might still be estimated by using project reports and some 
additional information which may be provided by the directors. 

Ten projects have continued, but most appear to be doing so 
with very minimal financial support. One aspect needing some 
clarification is how much of the original or intended program was 
cut due to lack of funds. It also needs to be determined how 
much is being maintained "out of- the hides" of dedicated 

20 



BEST COPY AVAILABLE 



directors who refuse to let their projects die. Such sacrifices 
could not be expected to transfer to most future initiatives. 

The wide variety of circumstances and results obtained from 
this set of pilot projects provides a rich potential source of 
information. The information from this survey will be further 
consolidated with project reports and other additional 
information in an attempt to "tease out" clues that might be 
utilized to direct future initiatives. The goal will be to 
provide policy makers with a clear, organized assessment of what 
has been successfully implemented, and a realistic picture of the 
obstacles they will face in their efforts to institutionalize 
programs of student outcomes assessment. 

RESULTS FROM THE ASSESSMENT OF MULTICULTURAL VALIDITY 

In the technical language of educational research, 
measurement accuracy is defined and understood as a matter of 
test validity. Consistent with the criteria more generally 
stated above, a valid test is one that provides an adequate means 
for making good inferences. When we develop a procedure for the 
purpose of assessing student outcomes, the goal is to obtain 
"scores" that will enable us to reach meaningful conclusions 
concerning the adequacy of current teaching and learning, provide 
direction for future interventions, and enable good decisions 
relevant to resource allocations within educational systems . 

The ability to derive the underlying "meaning" of these 
assessment "scores" from which we hope to make valid inferences 
is complicated by the question of whether or not scores from a 





21 



common test (or any evaluative process) can be interpreted to 
mean the same thing* for different individuals. Where individuals 
being measured are not homogeneous in reference to social 
experiences, language, values, and learning styles, can their 
abilities, attitudes, beliefs, or aptitudes be adequately 
assessed and interpreted using a common measure for all? This 
question is one of multicultural validity. 

Data Collection 

The diverse, multicultural environments of many of the 
schools within the California State University System provide an 
excellent field of study for the question of multicultural 
adequacy of assessment procedures. Data was collected from 
reports and articles presented or published by the project 
directors of experimental outcomes assessment studies. 

Additional information was obtained in a phone survey of 
directors. The structure used for this interview is attached in 
Appendix C . ■ 

It was apparent from this data that not all projects 
obtained demographic breakdowns of their data. This excluded 
them from contribution to this analysis. 

For those who had identified results by groups, several were 
hesitant to release specific information concerning the 
differences they had uncovered. Very few projects included group 
difference information in their final reports. The telephone 
interviews gathered enough information to make the analysis 
possible. However, some directors resisted releasing specific 
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statistics due to an expressed fear that such data would be 
misinterpreted as racist in motivation. 

But many differences did exist, not only in standardized 
test results, but in almost every type of outcomes assessment 
attempted. Awareness of these differences is a necessary first 
step in developing policy to address inequities. 

Validity Paradigm Used for Assessment 

The paradigm of measurement validity was useful because it 
provided a means of approaching and organizing a highly diverse 
set of data. By stepping through the conceptual stages of 
content, criterion, and construct validity, one first accumulates 
igola.ted pieces of relevant information concerning the nature of 
measures used and comparative assessment outcomes obtained from 
all projects (content analysis) . Once these are described and 
potentially problematic areas are identified, the process 
continues with attempts to better understand these measures by 
evaluating pairwise associations between these assessment 
variables and other variables measured within the same study 
(criterion analysis). Finally, an attempt is made to build a 
logically consistent structure of relationships that seem to 
generalize across studies (construct analysis). 

All attempts at defining validity contribute to the same 
goal. One wishes to clarify the meaning of the measure so that 
better inferences might be made on the basis 01 the scores. For 
assessment scores, we need to better understand what individual 
attributes are represented by or contribute to the score. 
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will enable more accurate conclusions concerning the 
multicultural adequacy of the measures. The three stages of 
validity determination and their genaral purpose are summarised 
in Figure 2 . 

Multicultural Content Valid it_y 

Utilizing the issue of measurement validity as the guiding 
principle for this evaluation, the first step ^as to address the 
simple question of content validity. Classically defined, 
content validity refers to the degree to which the assessment 
procedures represent the appropriate content of what we wish to 
measure, and whether the scoring system for individual 
proficiency is correspondent with what it takes to score high or 
low* on the variable in question. In reference to our 
multicultural concern, the issue must be expanded to include an 
evaluation of the appropriateness of the assessment s content for 
all students subjected to that assessment. 

For example, scores from timed, multiple-choice tests are 
often used to assess outcomes of content-specific programs. 

Though it may not be made explicit, reading and comprehension of 
English are an integral part of the ’content of such a test. It 
is easy to see how this part of the test’s content 
d i s ad van t age s non-native English speakers, w*ith differential 
performance the result. 

Accordingly, the first "flagging’' device we used for 
diagnosing the multicultural validity of assessment procedures in 
these projects was the simple identification of differences among 
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groups on assessment scores. The logic of this step in reference 
to content validity is that significant group differences in 
assessment outcomes indicate assessment procedures that are 
problematic for certain groups. These mean differences across 
groups could indicate assessment bias or valid discrepancies in 
underlying achievement. 

The simple identification of group differences provides very 
little information about the causes or meanings of the 
differences in scores, but it spotlights circumstances which 
require further investigation. It must also be recognized that 
an absence of group differences on an assessment does not 
guarantee that these scores can be interpreted to mean the same 
for all groups, but such circumstances seem to warrant less 
immediate concern. 

Some of the key outcome assessment differences that were 
noted from the pilot projects are listed in Figure 3. These 
■included group differences on standardized tests, other 
objectively measured variables, and some differences that were 
subjectively noted by project directors. 

Multicultural Criterion Validity 

The second sweep at interpreting the multicultural adequacy 
of our assessment instruments was based on the general approach 
of criterion validity. This is a question of how well scores 
correspond to or predict scores on another variable. Where group 
differences exist, a deeper understanding of the meaning of these 
group differences may be obtained by evaluating the nature of the 
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pairwise relationships between the assessment scores and other 
variables . 

For example, if one knows the number of years that a student 
has spoken English, and this variable is highly predictive of the 
scores on the outcome assessment, this relationship provides some 
insight to the appropriate interpretation of the assessment 
score. A non-relationship is potentially just as informative. 

For instance, an assessment score which fails to predict post- 
graduate performance for a certain group of students immediately 
calls to question the multicultural validity of that assessment 
measure. 

In the projects studied, examples of several interesting 
(and often recurring) pairwise relationships were reported. Some 
of these are noted in Figure 4. 

Multicultural Construct Validity 

■ Ultimately, one must attempt to compose the "big picture." 
Construct validity provides the model for the depth of 
understanding we must strive to attain. Developing construct 
validity for an outcome measure enables the clarification of the 
meaning of the measure, understanding of how it relates to other 
previous or concurrent events/variables, and the use of the , 
measure to predict subsequent behaviors or attitudes. 
Multicultural construct validity is complicated by the fact that 
these three aspects of the measure may vary by group. 

We cannot claim to have developed a comprehensive definition 
of the system (i.e., " nomolog ical net") within which measures of 
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student outcomes exist. In fact, this "net" will most certainly 
vary somewhat by content area; nevertheless, we believe our 
evaluation provides evidence of some consistent structural 
patterns. The major pieces of this structure are presented in 
Figure 5 . 

In the California State University studies, language 
background, basic learning skills, and climate in the major 
appear to be key initial variables in the model. These were 
clearly predictive of eventual attainment of skills in the major, 
standardized test scores, GPA , and attrition. In trun, these 
collective undergraduate experiences are associated with terminal 
outcomes such as perceived value of the major, quality of the 
degree program, and occupational prestige. 

Summary 

This model provides the basis for a very preliminary- 
diagnosis of multicultural validity. The evidence suggests that 
the meaning of certain measures observed may potentially differ 
by group. Basic language skills are highly predictive of many 
types of assessment scores, and, consequently, must be confounded 
with what these assessment procedures are attempting to measure. 
Differences by groups in the magnitude of relationships between 
assessment scores and subsequent outcomes ( in work or additional 
schooling) would further suggest that measures may successfully 
tap underlying achievement in some students, but not for others. 

A key variable contributing to the final description of the 
relative success or failure of student outcomes assessment in the 
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California State University system will be the ability of the 
data collected ' from these studies to define the adequacy of our 
educational procedures/environments for all groups of students 
represented within the system. The results of this initial 
investigation of multicultural validity must be considered 
preliminary at best. Only continued accumulation of data will 
enable any final conclusion doncerning the adequacy of assessment 
procedures. Can we safely infer that our assessment tools 
provide meaningful information about students from all groups. 
This is the question that must direct the selection of measures, 
the development of experimental hypotheses, and the subsequent 
addition of pieces to the puzzle of multicultural validity. 
RESULTS FROM SUMMARY ASSESSMENT OF GENERALTZABILITY OF VARIABLES 
ACROSS PROJECT SITES 

Data organization was achieved in a manner consistent with 
the recommendations for qualitative data analysis prescribed by 
Miles and Huberman (1984). All observations, comments, and 
survey scores were condensed and entered in a highly abbreviated 
form onto a "meta-matrix." This master chart contains 
information relevant to all thirty— seven variables for all 
projects considered complete enough for inclusion in the final 
assessment. In early versions of the chart, the basic principle 
was inclusion of all relevant data. A small, condensed sample 
illustrating the layout of this meta— matrix is attached in 
Appendix D. Actual data cells were much mere comprehensive. 

For this final analysis, projects for which we lacked 
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assessed three 



comprehensive data were excluded. One project 
different disciplines on three different campuses. Because the 
five sources providing information for this analysis were 
describing discrete event from what were apparently very 
different experiences, these data sources were treated as five 
different "sites” for this analysis. The total 'n of sites 
ultimately used was sixteen. 

On the basis of information represented in this matrix, a 
qualitative categorization of all projects on all variables was 
completed. For each project, all variables were classified as: 

4 - strongly present/achieved, 

3 - partly present/achieved, 

2 - weakly present/achieved, 

1 - absent/not achieved. 

The results of this classification are presented in Table 2. 
This display is referred to by Miles and Huberman (1984) as a 
"Site-Ordered Descriptive Meta-Matrix." Though the "values” are 
the result of qualitative assessment and therefore include a 
certain unavoidable level of judgement, almost all are based on 
multiple sources of information. The agreement between sources 
was generally very compelling. 

Using the proposed research conceptualization, project 
outcomes (direct and indirect) would logically represent 
dependent measures. Environmental and methodological variables 
would be the multiple predictors. If the data were interval, and 
the n of projects greater, the analysis of choice would be 
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multiple regression. In a multiple regression paradigm, critical 
factors that should be attended to in any replication would be 
those that accounted for the most variance in the dependent 
measure ( program outcomes ) . 

However, the data presented in Table 2 is qualitative, and 
the n of projects is sixteen. Nevertheless, the problem is 
conceptually the same. One must identify the variables that were 
consistently associated with project effectiveness. The task is 
to determine which predictor variables most consistently co _, »ary 
with outcomes. Good predictors are those whose qualitative 
categorization is most consistent with the qualitative 
categorization of project outcomes. 

Because a project’s relative success should not be judged by 
a single outcome variable, and because there should be some 
resistance to the temptation to ove reanalyze qualitative data, 
outcomes were combined. Subsequent analyses were conducted 
using: 1) the average of all outcomes, 2) the average of outcomes 

categorized as direct outcomes, and 3) the average of indirect 
outcomes (refer to page 6 for a precise listing of these 
outcomes ) . 

The next step necessary to enable comparison across 
variables is a standardizat ion of scores. Though the ’scale 
used for classification of all predictor and outcome variables 
was the same (1 through 4), the mean and standard deviation of 
the resulting distributions for each variable was different. 

This was resolved by transforming all scores to z-scores. 
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resulting' standardized site-crdered descriptive meta-matrix is 
attached in Appendix E. These scores represent the project’s 
deviation above or below each given variable’s mean in standard 
deviation units. Scores from all variables are now represented 
in reference to their place on a common distribution that has a 
mean of zero and a standard deviation of one. 

"Consistency", between predictors and outcomes was assessed 
by computing the squared deviations between each standardized 
score and the average "outcome” (overall, direct, &. indirect) for 
the project, associated with that standardized score. The squared 
deviations between each standardized predictor and the overall, 
direct and indirect outcomes are attached in Appendices F, C-, AND 
H. 

Finally, these squared deviations were summed across project 
sites, and divided by the number of projects to produce a value 
conceptually similar to variance (the difference being that the 
source of deviation was between each predictor score and its 
project’s outcome score rather than between each predictor score 
and the average of that- predictor score across sites). The 
resulting "variance with outcomes” across sites will be smelliest 
for those variables that were most consistently related to 
outcomes. Larger values indicate that the classifications 
assigned to that variable were not as predictive of the project 
outcomes . 

Results of this analysis are presented in Tables 3, 4, and 

5. Because the data is qualitative, there can be no assessment 
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of statistical probability or significant differences. The 
predictor variables are simply ranked in the order of their 
consistency with outcomes: overall, direct, and indirect. 

Top-ranked variables are those whose classifications were most 
consistent with project outcomes. 

These results are summarized in Table 6. This table enables 
comparison of the relative importance of predictors across 
different combinations of project outcomes. Predictors are also 
organized into groups by similarity. 

ENVIRONMENTAL PREDICTORS 
Faculty Involvement Variables 

Two faculty variables were consistently tow’ard the top of 
the respective rank orderings. "Faculty Participation and 
"Perceived Faculty Workload" appear to be potentially important 
indicators of assessment success. The first is most closely 
associated with those outcomes identified as direct outcomes. 
The second is more closely associated with "indirect" outcomes. 

"Faculty Ownership" and "Faculty Consensus with Plan" were 
also relatively good indicators, especially for direct outcomes. 
"Faculty Participation in Planning" appears to have little to do 
with project outcomes. 

Training/Experience with Student Outcomes Assessment 

Project outcomes were associated with the project director s 
efficacy in outcomes assessment, especially indirect outcomes. 
Faculty training and the project director* s general academic 
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Table 3: Predictor Variables Rank Ordered by Consistency with 

Overall Outcomes 



Predictor Variables 

Adequacy of Measures Dev. 
Intended Breadth of Aud . 
Administrative Support 
Faculty Part, in Project 
Director’s Training in SOA 
Perceived Faculty Workload 
Sensitivity to Mult Issues 
Development of Mult Measures 
Selection of Outcomes 
Measurement Properties 
Statistical Analysis 
Faculty Consensus with Plan 
Project Goal Definition 
Faculty Ownership of Project 
Data Collection/Reporting 
Existence of Previous SOA 
Faculty Training in SOA 
Student Support of Project 
Adequate Budget/Supplies 
Content Domain 
Report Comprehensiveness 
Utility/Economy of Project 
Director’s Academic Exper. 

Fac Participation in Planning 



Variance with 
Overai 1 Outcomes 



1 


. 73 


2 


. 7 5 


3 


.76 


4 


. 7 7 


5 


.91 


6 


. 9 5 


7 


1 .06 


8 


1 . 10 


9 


1 .12 


10 


1 . 22 


11 


1 . 25 


12 


1.27 


13 


1 . 28 


14 


1 .31 


15 


1 . 39 


16 


1 . 50 


17 


1.60 


18 


1 . 60 


19 


1 .71 


20 


1 .79 


21 


1.80 


22 


2 .32 


23 


2.41 


24 


2.43 
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Table 4: Predictor Variables Rank Ordered by Consistency with 

Direct Outcomes 



Predictor Variables 



Rank Order 



Variance with 
Direct Outcomes 



Faculty Part, in Project 
Adequacy of Measures Dev. 
Admin i st rat i ve Support 
Faculty Consensus w'ith Plan 
Intended Breadth of Aud . 
Development of Mult Measures 
Faculty Ownership of Project 
Selection of Outcomes 
Perceived Faculty Workload 
Sensitivity to Mult Issues 
Student Support of Project 
Director’s Training in SOA 
Measurement Properties 
Statistical Analysis 
Project Goal Definition 
Faculty Training in SOA 
Data Collect ion/ Re porting 
Content Domain 
Existence of Previous SOA 
Adequate Budget /Supplies 
Utility/Economy of Project 
Report Comprehensiveness 
Director’s Academic Exper. 

Fac Participation in Planning 



1 


. 5 5 


2 


. 80 


3 


.89 


4 


1 .00 


5 


1 .09 


6 


1 .09 


7 


1 . 23 
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1 .23 


9 


1 .25 


10 


1 .27 


11 


1.31 


12 


1 .32 


13 


1 .37 


14 


1 .48 


1 5 


1 . 54 


16 
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17 


1 . 64 


18 


1 .83 


19 
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20 
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Table 5: Predictor Variables Rank Ordered by Consistency with 

Indirect Outcomes 



Variance with 



Predictor Variables 


Rank Order 


Indirect Outcomes 


Intended Breadth of Aud . 


1 


. 66 


Director’s Training in SOA 


2 


.72 


Perceived Faculty Workload 


3 


.86 


Administrative Support 


4 


. 39 


Adequacy of Measures Dev. 


5 


. 92 


Sensitivity to Mult Issues 


6 


1 . 06 


Project Goal Definition 


7 


1.17 


Selection of Outcomes 


8 


1.17 


Statistical Analysis 


9 


1 . 20 


Faculty Part, in Project 


10 


1.22 


Measurement Properties 


11 


1.24 


Existence of Previous SOA 


1 2 


1.25 


Development of Mult Measures 


13 


1 .29 


Data Collection/Reporting 


14 


1*36 


Report Comprehensiveness 


15 


1.44 


Faculty Ownership of Project 


16 


1.53 


Adequate Budget/Supplies 


17 


1 . 62 


Faculty Training in SOA 


18 


1.66 


Faculty Consensus with Plan 


19 


1 .69 


Content Domain 


20 


1.80 


Student Support of Project 


21 


1 . 96 


Fac Participation in Planning 


22 


2.32 


Director’s Academic Exper. 


23 


2.33 


Utility/Economy of Project 


24 


2 .49 
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Table 6: Variance between Predictors and Outcomes with Predictor 

Variable Rank Orders* 



Varnce w*ith Varnce with Varnce with 
Predictor Variables All Outcomes Dir Outcomes Inc Outcomes 



Fac Fart in Planning 2.43 
Faculty Part, in Project 0.7/ 
Fac Ownership of Project 1.31 
Fac Consensus with Plan 1.27 
Perceived Faculty Workload 0.S5 

Director’s Training in SO A 0.91 
Faculty Training in SOA 1.60 
Director’s Academic Exper. 2.41 

Adequate Budge t /Suppl ies 1.71 
Administrative Support 0.76 
Student Support of Project 1.60 

Existence of Previous SOA 1.50 

Content Domain 1.79 
Intended Breadth of Aud . 0.75 

Project Goal Definition 1.28 
Selection of Outcomes 1.12 
Adequacy of Measures Dev. 0.73 
Data Collection/Reporting 1.39 
Measurement Properties 1.22 
Statistical Analysis 1.25 

Sensitivity to Mult Issues 1.06 
Develpmnt of Mult Measures 1.10 
Report Comprehensiveness 1.80 

Utility/Economy of Project 2.32 
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background and experience were not as closely tied to project 
results. 

Support Variables 

Administrative support was closely tied to project outcomes, 
both direct and indirect. Adequacy of budget, supplies, and 
other institutional resources was not closely tied to project 
results. Student support appears only moderately associated with 

direct outcomes. 

Existing Student Outcomes Assessment Procedures 

Whether or not formal programs for or methods of student 
outcomes assessment existed prior to the initiation of the pilot 
projects was not closely associated with project results. Prior 
experience with assessment did not seem to have a consistent 
positive or negative effect on direct or indirect outcomes. 
Project Focus 

The content area in which the project occurred had little to 
do with the relative success of the project. However, the 
intended breadth of the audience (who and how mank- 
ind ividual s/ o rgani zat ion I expect to learn about the results of 
mv project) was very closely associated with outcomes, especially 
indirect outcomes. 

General Procedural Adequacy 

The single most critical variable from this category was the 
development of good measures of student outcomes. The creation 
of good measures was predictive of project results, direct and 
indirect. Appropriate selection of outcomes to measure was also 
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closely associated with project outcomes. 

"Project Goal Definition and the adequacy of "Statistical 
Analyses" conducted were moderately tied to indirect outcomes. 
These variables along with "Data Collection/Reporting" and the 
reporting of "Measurement Properties" did not appear to co-vary ■ 
tightly with project results. 

Project Comprehensiveness 

The "Development of Multiple Measures" of student outcomes 
and "Sensitivity to Multicultural Issues" appeared moderately 
associated with project outcomes. The comprehensiveness of the 
reports made available were not closely related to project 
outcomes . 

Cost Effectiveness of Project 

The "Util i ty/Ec onomy " of the projects’ assessment procedures 
was near the bottom of the rank ordering. Apparently expensive 
projects in terms of their dollars spent to students assessed 
were not always the richest in results. 

Summary 

It was expected that faculty involvement would be critical 
to project outcomes. Consequently, it was no surprise that 
faculty participation in the project was. ranked number one in its 
association w r ith the direct, intended outcomes of each project. 
Faculty consensus with the project plan, ownership of the 
project, and perceived workload were also ranked in the top ten 
in reference to direct outcomes. 

Perceived vrorkload appeared especially critical to indirect 



outcomes. Perhaps as workload increases, it becomes less likely 
that indirect outcomes will be realized. Workload would also 
seem especially salient to the development of attitudes toward 
outcomes assessment (one of the indirect outcomes). 

Somewhat surprising was the apparent unimportance of 
involving participating faculty in the early stages of project 
planning. In the general management literature, it is often 
suggested that participative planning of projects is a good 
strategy for insuring project participation. Apparently in the 
academic setting, it is only critical that you do an adequate job 
of "selling" your plan once it is developed. 

Of the three factors describing the training and background 
of project participants, only the project director s training and 
experience specific to student outcomes assessment ranked very 
high in association with outcomes. This factor was especially 
critical to the attainment of indirect outcomes. This result is 
logical given that the director was generally responsible for the 
adequacy of project goals and methods. 

Nothing happens without administrative support. In all 
hierarchies, policy tends to come from the top down. Though a 
goal of the CSL' system is to establish f acuity- init iated outcomes 
assessment, these initiatives will most likely fail without the 
support of academic administrators. These individuals are in the 
position to set the "tone" in reference to the academic 
legitimacy of such efforts. Unless outcomes assessment efforts 
are rewarded, it will be difficult to maintain motivation to 



assess student outcomes. 



Student support did not rank high. i hough this source Ot 
support was assessed through the eyes of tne project directors 
rather than from the students themselves, this result does not 
appear unreasonable. 

What does seem somewhat surprising is the apparent 
unimportance of adequate budget and supplies. Some insight into 
this result was obtained in discussions with project directors 
about how* much of their project was coming out of their hide. 

The efforts of some directors clearly went beyond what might have 
been expected given the modest budgets they were receiving for 
their administration of each project. This phenomenon was more 
likely to occur if the director was working in a content area for 
which outcomes assessment research could be considered legitimate 
professional development or for senior faculty members who were 
no longer struggling to achieve tenure or promotion. 

Another possible explanation for the inconsistency between 
the adequacy of budget/supplies and project success would be the 
inherent differences in requirements for adequate assessment 
procedures across different content areas. It may be possible 
that adequate assessment can be achieved for relatively- small 
costs in some disciplines, but will tend to be verv expensive in 
others. Consequently, adequate results might be obtained j.or 
some even when resources are tight while others will find 
assessment prohibitive without adequate budgetary support. 

Whether or not the project was initiated in an environment 
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that currently included some type of formal outcomes assessment 
did not appear to have a critical impact on project outcomes. 

The good news of this result is that projects breaking new ground 
do not necessarily have to anticipate damaging levels cf 
resistance. The bad news may be that previous experience with 
outcomes assessment apparently may not guarantee that new 
initiatives will be welcomed with open arms. 

There was some expectation that perhaps the content area in 
which the project occurred might have considerable impact on the 
relative success of the project. This may disappoint some 
educational or behavioral science researchers who. might have 
liked to as sume they’ had a corner on this part of the research 
market. This is a good result for a system that hopes to 
initiate outcomes assessment across a universe of content 
domains. 
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that the excellence of project outcomes should be related to how- 
many individuals or organizations with which the director intends 
to share the results. This result may also reflect the 
director’s experience and enthusiasm for student outcomes 
assessment. As previously observed) project director efficacy in 
outcomes assessment was closely associated w'ith project outcomes. 

In reference to general procedural adequacy') the most 
important aspect of the assessment projects appeared to 

46 



ERIC 



VS 



be the 



adequacy of the measures they developed or adopted. It is highly 
logical to expect that project success would hinge on the ability 
of the measures used to reliably and validly measure student 
outcomes. This process begins with the selection of appropriate, 
outcomes to measure. This aspect of procedural adequacy was also 
closely associated with project outcomes. 

Other variables in this category, though not highly ranked 
in their association with direct outcomes, were toward the top of 
the list in relation to indirect outcomes. Much of this 
relationship appeared dependent upon the association with the 
indirect outcome of external adoption. ihose projects bound for 
adoption were generally the most precise in the definition of 
project goals and most ambitious and accurate in the production 
of statistical analyses. 

The relatively high ranking of "sensitivity to multicultural 
issues" and the development of multiple measures may again 
reflect the sophistication of the project director in reference 
to good outcomes assessment. It is also reasonable to expect 
projects that developed or used more than one form of assessment 
to be more successful. 

Sensitivity to multicultural issues would appear especially 
relevant to the potential for recruitment and retention of under- 
represented students. This would partially explain this 
variables relatively high ranking in reference to indirect 
outcomes. 

The nature of final project reports varied widely. The low 
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project reports and 



association between the comprehensiveness of 
project outcomes might indicate that much occurred that was not 
completely reported. Many directors indicated that deadlines 
fell before they had time, to adequately process the project 
results. Some compensated by disseminating results via other 
channels (regional and national presentations, journal 
publications). Though this project followed up and received many 
of these reports, it appears that much information from 
successful, comprehensive projects was lost to the CSL' system due 
to the fact that final reporting was required before assessment 
procedures could be adequately evaluated. 

The cost-effectiveness of outcomes assessment was near the 
bottom of the list in its association with direct and indirect 
outcomes. This is a perfectly reasonable result given the 
experimental nature of these projects. Directors "tried out” a 
wide variety of assessment procedures. There will also always be 
some cost differences across disciplines necessitated by the 
different nature of assessment processes that must occur. Some 
content areas may be able to conduct excellent cheap 
assessment, while other areas may only achieve moderate results 
despite a high price tag. 

Nevertheless, there is some logic to the assumption that 
efficient assessment will produce more desirable results. As 
more data is collected, this relationship should be reassessed 
within content areas. 

Though qualitative analysis does not enable the same level 
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of precision as might be obtained from more Quantitative data, 



the observations made in this study were systematic, relatively 
objective, and almost always based on multiple sources. 

Additional testing and replications of this study’s conclusions 
is suggested, and could occur within the CSL' ’ s continuing program 
to develop and monitor programs of student outcomes assessment . 

To briefly summarize, the results of this analysis suggest 
that future initiatives be especially sensitive to four 
variables: 1) faculty participation in the project, 2) adequacy 

of the measures developed/adopted, 3) development of 
administrative support, and 4) the assessment director’s 
t r a i n i ng / expe r i ence with student outcomes assessment procedure. 
MAJOR CONCLUSIONS 

The major conclusions drawn from an overview of the 
project’s results can be organized in terms of the conceptual 
model: environmental variables and methodological variables as 

they relate to outcomes. 

Environmental Variables 

The environmental variables critical to project outcomes 
were human resources. First, recruiting and maintaining faculty 
support was a key vari ab le, but one which showed high variability 
across projects. The factors underlying this variability are 
difficult to detect in the quantitative measures, but they appear 
in the interview data from project directors. One concern in 
nearly .all faculty groups is "the intended primary use of the 
outcomes data," particularly where data suggest evidence of 
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teaching/program effectiveness. Another might 
the worry over the human capital costs of department-level 
assessment activities. This turned up as a particular concern 
for junior faculty. Many junior faculty perceive that research 
on teaching and learning is regarded as "second-tier" research 
which may not be counted in the tenure/promotion, process. As 
several respondents indicated, assessment activities favor two 
faculty groups: 1) those in social/behavioral disciplines, and 2 

those whose professional research activities fit with 



assessment, research. 

Drawing from the experiences of the project directors, ther 
appear to be several general guidelines for establishing and 
maintaining faculty participation: 

1) Educate participants about the value of assessing 
student outcomes. The motivation required to commit to 
outcomes assessment is dependent upon a general 
perception that this effort will provide payoffs. The 
general descriptions of outcomes attained (refer to 
survey results) provide more than adequate examples of 
the benefits achieved by effective programs. On an 
individual basis, participation will be enhanced if 
assessment activities are an integral part of the 
faculty performance criteria. 

2) Maintain local control of the project. Support was 
never achieved in one project partly because the 
program was perceived as "someone else’s grand plan 
that got shoved down our throats. Support for another 
project was in jeopardy when the local faculty senate 
returned the project’s proposal as an external mandate. 

3) Overcome the "threat” associated by many faculty 
with assessing outcomes. Many directors reported the 
suspicion expressed by their faculty co n c erning the 
intended use of the data collected. Faculty often saw 
the potential for information obtained to be used as a 
club to punish programs or individuals rather than as a 
constructive tool for self —development • 
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The importance of the development of administrative support 
was cons istenly reported across projects. Data from the majority 
of respondents suggested that "in-kind'' resources, publicity, 
campus-level coordination, and establishing a climate receptive 
of assessment initiatives were important positive contributions 
of administrative offices. Even in this positive environment 
however, data indicate that more concrete evidence of integration 
of the assessment agenda in campus-level policy and in concrete 
recognition of assessment activities for professional development 
are needed. 

Two general prescriptions can be made on the basis of 
director responses. The first deals with the establishment of 
administrative support, the second partially defines the nature 



of the support sought: 

1) Educate administrators about the value of assessing 
student outcomes. Just as executives of business 
organizations value economic indicators for their 
companies, educational administrators need to 
understand the potential value of performance feedback 
inherent in the assessment of student outcomes. They 
must also be sensitized to the need for discretionary, 
constructive use of such data. The validity of 
departmental reports will quickly erode if messengers 
delivering "bad news" are shot. 



2) Adminis 
development 
legitimate 
reported th 
assessment 
faculty who 
term ’’waste 
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trators should recognize and reward the 
of outcomes assessment in one’s field as 
professional development. Many directors 
at contribution of time to outcomes 
was unrealistic for ail but senior, tenured 
could afford to "waste some time." The 
" was used only in reference to the 
or recognition of these efforts by chairs, 
department/ schco 1/ university evaluation 
in control of the retention, promotion and 
ess . 



Finally, project directors’ training/experience in 
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measurement and analysis was key to pr 
here there were important differences, 
reported dismay over the dirficulties 
procedures as the project progressed, 
echoed the sentiments of one director 
lost momentum f, just as experience and 



eject effectiveness, and 
Some project directors 
in learning assessment 
A number of respondents 
who felt that the project 
proficiency began to 



develop . ” 

These observations clearly suggest the importance of 

training in outcomes assessment for those administrating 

assessment development. Two general prescriptions flow from the 

comments made by project leaders: 

1) Assess the training/experience of project directors 
specific to assessing student outcomes. General 
knowledge and experience as an educator is not enough. 

This report’s "assessment" of relevant knowledge was 
dependent upon self-report. This approach appeared 
satisfactory, producing large variance in the levels of 
described outcomes assessment expertise* 



2 ) Provide continued opportunities for training in and 
exposure to outcomes assessment. Many of the directors 
reported that the system— wide conferences and reports 
on outcomes assessment had been responsible for 
kindling their interest in the process. If faculty- 
based outcomes assessment is to be "institutionalized’ 
within the CSU, we must continue to share knowledge. 

Methodological Variables 



The key methodological variable was the development or 
adoption of adequate measures of student outcomes. Adequacy of 
measurement implied several aspects. The first is the simple 
psychometric properties of the assessment. This involves the 
reliability and validity of the assessment procedures. As an 
example of awareness of measurement adequacy, several directors 
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reliability of 



did an excellent job of assessing the inter-rater 
judges producing qualitative assessments or student projects or 
papers. Others spent considerable time and consulted widely with 
their peers to evaluate the content validity o x their 
assessments. This process often had positive, retroactive impact 
on curriculum and teaching strategies. 

Multicultural sensitivity also contributed to the adequacy 
of measurement, especially in reference to the inferences drawn 
from assessment scores. A critical question for directors to ask 
was, "what assessment procedures will provide all students viith 
an equitable opportunity to demonstrate their 'competence; This 
consideration should result in the production of multiple, more 
creative indices that would provide a more comprehensive picture 
of student achievement. 

Multiple types of assessment also enabled directors to 
obtain feedback on more than one type of outcome. Rather than 
focusing solely on content or cognitively-based outcomes, 
additional measures of affective and at t itudinal variables 
resulted in a much richer, more complex basis for judgements of 
program adequacy. 

The bottom line is that measures cannot be haphazardly 
developed or selected. Future faculty initiators of student 
outcomes assessment should not be told if, when, and what to 
assess, but most may benefit from technical support in reference 
to how to asssss . 

The following summary prescriptions are derived from 
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observations related to procedural and measurement adequacy: 

1) Clearly define educational objectives. The nature 
of the assessment tools cannot be determined until the 
desired outcomes are described. Several directors 
reported that healthy re-evaluations of their 
curriculum and program goals were a necessary precursor 
to the development of the actual assessment 
instruments . 

2) Assess multiple outcomes. Directors indicated that 
since educational objectives were seldom 

unidimens ional , it made little sense to attempt to 
assess educational criteria with a single measure. The 
richest data sources enabling the clearest assessment 
of program outcomes involved combinations of content 
tests, performance— based demonstrations, attitude 
assessments, affective measures, etc. 

3 ) Sensitivity to test fairness across constituent 
groups. Not all directors dealt with this issue, but 
those who did provided ample evidence of differential 
performance across groups. Though some differences may 
validly reflect the results of disadvantaged 
preparation for higher education, assessment procedures 
must minimize performance deficits related to native 
language differences and/or content that contains 
cultural/soc ioeconoinic bias. 

4) Assessment of measurement reliability/validity. 

The importance of this recommendation cannot be over- 
emphasized. If an assessment instrument does not 
possess adequate psychometric properties, it provides 
no basis for meaningful inference concerning the 
relative performance of the student or the success of 
the academic program. No amount of -faculty and 
administrative support will not save a project based on 
faulty measurement. 

Outcomes 

The adequacy of project outcomes to provide direction for 
future assessment initiatives is a critical dimension of project 
effectiveness. Evidence for many of the direct and indirect 
outcomes anticipated in the model for the study proved difficult 
to evaluate, in part -because some of the target outcomes are Ion 
-term effects which would require longitudinal measurement. In 
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addition, some data on outcome variables were still being' 
collected as project final reports fell due, so did not get 
included as outcomes. 

The outcome which may be the best index of project 
effectiveness appeared to be dissemination of results. Project 
directors’ verbal reports consistently underscored the importance 
cf information sharing as an outcome of project involvement. 
Moreover, the projects with explicit descriptions of plans for 
dissemination to an identified audience in their early goal 
definition continue to be active in publishing results. Further 
study of these projects and their results should provide 
additional direction for future strategies in faculty- ini t iated 
assessment programs. 
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APPENDIX E 



"TOWARD 

10:00 - 10:30 
10:30 - 12:00 

12:00 - 1:30 
1:30 - 2:15 

2:15 - 3:00 

3:00 - 3:15 



California State University 
Institute for Teaching and Learning 



1991 CSU Systemwide Seminar on 
Student Outcomes Assessment 

NEW PARADIGM: ASSESSMENT IN MULTICULTURAL CALIFORNIA" 

June 8, 1991 

San Francisco Hilton Hotel 



Seminar Agenda 



Opening & Remarks 

Olita D. Harris, Seminar Chair and Coordinator 
San Diego State University 

Sherrin Marshall, Program Officer 

Fund for the Improvement of Postsecondary Education 

Results of the Evaluation Study 

Matt Riggs, Assistant Professor 
California State University, San Bernardino 

Joanna Worthley, Assistant Professor 
California State University, San Bernardino 



Lunch (on your own) 



Assessment & the Public Policy Agenda: Shifting Sands 
Peter Ewell, Senior Associate 

National Center for Higher Education Management Systems 
Whither WASC Goeth 

Ralph Wolff, Associate Executive & Director 
Western Association of Schools and Colleges 

Break 



3:15 - 4:25 



Promising Practices - Small Group Discussions 



Subject Matter Asses sment in Teacher E ducation 
Convener: Bernice Bass de Martinez 

Chair, School of Education & Human Development 
CSU, Fresno 

Multicultural Issues in General Education/Transfer 
Convener: Cynthia Flores 

Assistant Dean for Student Affairs 

San Diego State University, Imperial Valley Campus 

ESL Assessment 
Convener: Roberta Ching 

Assistant Professor of Learning Skills 
CSU, Sacramento 

Assessing Discrimination & Reverse Discrimination 
Convener: Kenneth Nyberg 

Professor of Anthropology and Sociology 
CSU, Bakersfield 

Critical Thinking Assessment 
Convener: Susan Nummedai 

Professor of Psychology and Chair of 

CSU Critical Thinking Council Assessment Group 

CSU, Long Beach 

Reading and Writing Competence: Implications for Multicultural 
General Education 

Convener: Priscilla Chaffe-Stengel 

Professor of Information Systems and Decision Science 
CSU, Fresno 

4:30 - 5:40 Repeat Promising Practices - Small Group Discussions 

5:45 - 6:00 Wrap-Up 

Frank Young 

Statewide Director 

California Academic Partnership Program 
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